{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation and hyperparameter tuning\n",
    "\n",
    "In the previous notebook, we saw two approaches to tune hyperparameters.\n",
    "However, we did not present a proper framework to evaluate the tuned models.\n",
    "Instead, we focused on the mechanism used to find the best set of parameters.\n",
    "\n",
    "In this notebook, we reuse some knowledge presented in the module \"Selecting\n",
    "the best model\" to show how to evaluate models where hyperparameters need to\n",
    "be tuned.\n",
    "\n",
    "Thus, we first load the dataset and create the predictive model that we want\n",
    "to optimize and later on, evaluate.\n",
    "\n",
    "## Loading the dataset\n",
    "\n",
    "As in the previous notebook, we load the Adult census dataset. The loaded\n",
    "dataframe is first divided to separate the input features and the target into\n",
    "two separated variables. In addition, we drop the column `\"education-num\"` as\n",
    "previously done."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "target_name = \"class\"\n",
    "adult_census = pd.read_csv(\"../datasets/adult-census.csv\")\n",
    "target = adult_census[target_name]\n",
    "data = adult_census.drop(columns=[target_name, \"education-num\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our predictive model\n",
    "\n",
    "We now create the predictive model that we want to optimize. Note that this\n",
    "pipeline is identical to the one we used in the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import make_column_transformer\n",
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "from sklearn.compose import make_column_selector as selector\n",
    "\n",
    "categorical_columns_selector = selector(dtype_include=object)\n",
    "categorical_columns = categorical_columns_selector(data)\n",
    "\n",
    "categorical_preprocessor = OrdinalEncoder(\n",
    "    handle_unknown=\"use_encoded_value\", unknown_value=-1\n",
    ")\n",
    "preprocessor = make_column_transformer(\n",
    "    (categorical_preprocessor, categorical_columns),\n",
    "    remainder=\"passthrough\",\n",
    "    force_int_remainder_cols=False,  # Silence a warning in scikit-learn v1.6.\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import HistGradientBoostingClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "model = Pipeline(\n",
    "    [\n",
    "        (\"preprocessor\", preprocessor),\n",
    "        (\n",
    "            \"classifier\",\n",
    "            HistGradientBoostingClassifier(random_state=42, max_leaf_nodes=4),\n",
    "        ),\n",
    "    ]\n",
    ")\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "### Without hyperparameter tuning\n",
    "\n",
    "In the module \"Selecting the best model\", we saw that one must use\n",
    "cross-validation to evaluate such a model. Cross-validation allows to get a\n",
    "distribution of the scores of the model. Thus, having this distribution at\n",
    "hand, we can get to assess the variability of our estimate of the\n",
    "generalization performance of the model. Here, we recall the necessary\n",
    "`scikit-learn` tools needed to obtain the mean and standard deviation of the\n",
    "scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_validate\n",
    "\n",
    "cv_results = cross_validate(model, data, target, cv=5)\n",
    "cv_results = pd.DataFrame(cv_results)\n",
    "cv_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cross-validation scores are coming from a 5-fold cross-validation. So we\n",
    "can compute the mean and standard deviation of the generalization score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    \"Generalization score without hyperparameters\"\n",
    "    f\" tuning:\\n{cv_results['test_score'].mean():.3f} \u00b1\"\n",
    "    f\" {cv_results['test_score'].std():.3f}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now present how to evaluate the model with hyperparameter tuning, where an\n",
    "extra step is required to select the best set of parameters.\n",
    "\n",
    "### With hyperparameter tuning\n",
    "\n",
    "As shown in the previous notebook, one can use a search strategy that uses\n",
    "cross-validation to find the best set of parameters. Here, we use a\n",
    "grid-search strategy and reproduce the steps done in the previous notebook.\n",
    "\n",
    "First, we have to embed our model into a grid-search and specify the\n",
    "parameters and the parameter values that we want to explore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "param_grid = {\n",
    "    \"classifier__learning_rate\": (0.05, 0.5),\n",
    "    \"classifier__max_leaf_nodes\": (10, 30),\n",
    "}\n",
    "model_grid_search = GridSearchCV(model, param_grid=param_grid, n_jobs=2, cv=2)\n",
    "model_grid_search.fit(data, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As previously seen, when calling the `fit` method, the model embedded in the\n",
    "grid-search is trained with every possible combination of parameters resulting\n",
    "from the parameter grid. The best combination is selected by keeping the\n",
    "combination leading to the best mean cross-validated score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_results = pd.DataFrame(model_grid_search.cv_results_)\n",
    "cv_results[\n",
    "    [\n",
    "        \"param_classifier__learning_rate\",\n",
    "        \"param_classifier__max_leaf_nodes\",\n",
    "        \"mean_test_score\",\n",
    "        \"std_test_score\",\n",
    "        \"rank_test_score\",\n",
    "    ]\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_grid_search.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One important caveat here concerns the evaluation of the generalization\n",
    "performance. Indeed, the mean and standard deviation of the scores computed by\n",
    "the cross-validation in the grid-search are potentially not good estimates of\n",
    "the generalization performance we would obtain by refitting a model with the\n",
    "best combination of hyper-parameter values on the full dataset. Note that\n",
    "scikit-learn automatically performs this refit by default when calling\n",
    "`model_grid_search.fit`. This refitted model is trained with more data than\n",
    "the different models trained internally during the cross-validation of the\n",
    "grid-search.\n",
    "\n",
    "We therefore used knowledge from the full dataset to both decide our model\u2019s\n",
    "hyper-parameters and to train the refitted model.\n",
    "\n",
    "Because of the above, one must keep an external, held-out test set for the\n",
    "final evaluation of the refitted model. We highlight here the process using a\n",
    "single train-test split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data, target, test_size=0.2, random_state=42\n",
    ")\n",
    "\n",
    "model_grid_search.fit(data_train, target_train)\n",
    "accuracy = model_grid_search.score(data_test, target_test)\n",
    "print(f\"Accuracy on test set: {accuracy:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The score measure on the final test set is almost within the range of the\n",
    "internal CV score for the best hyper-parameter combination. This is reassuring\n",
    "as it means that the tuning procedure did not cause significant overfitting in\n",
    "itself (other-wise the final test score would have been lower than the\n",
    "internal CV scores). That is expected because our grid search explored very\n",
    "few hyper-parameter combinations for the sake of speed. The test score of the\n",
    "final model is actually a bit higher than what we could have expected from the\n",
    "internal cross-validation. This is also expected because the refitted model is\n",
    "trained on a larger dataset than the models evaluated in the internal CV loop\n",
    "of the grid-search procedure. This is often the case that models trained on a\n",
    "larger number of samples tend to generalize better.\n",
    "\n",
    "In the code above, the selection of the best hyperparameters was done only on\n",
    "the train set from the initial train-test split. Then, we evaluated the\n",
    "generalization performance of our tuned model on the left out test set. This\n",
    "can be shown schematically as follows\n",
    "\n",
    "![Cross-validation tuning\n",
    "diagram](../figures/cross_validation_train_test_diagram.png)\n",
    "\n",
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p>This figure shows the particular case of <strong>K-fold</strong> cross-validation\n",
    "strategy using <tt class=\"docutils literal\">n_splits=5</tt> to further split the train set coming from a\n",
    "train-test split.\n",
    "For each cross-validation split, the procedure trains a model on all the red\n",
    "samples, evaluates the score of a given set of hyperparameters on the green\n",
    "samples. The best hyper-parameters are selected based on those intermediate\n",
    "scores.</p>\n",
    "<p>Then a final model tuned with those hyper-parameters is fitted on the\n",
    "concatenation of the red and green samples and evaluated on the blue samples.</p>\n",
    "<p class=\"last\">The green samples are sometimes called a <strong>validation sets</strong> to differentiate\n",
    "them from the final test set in blue.</p>\n",
    "</div>\n",
    "\n",
    "However, this evaluation only provides us a single point estimate of the\n",
    "generalization performance. As you recall from the beginning of this notebook, it is\n",
    "beneficial to have a rough idea of the uncertainty of our estimated\n",
    "generalization performance. Therefore, we should instead use an additional\n",
    "cross-validation for this evaluation.\n",
    "\n",
    "This pattern is called **nested cross-validation**. We use an inner\n",
    "cross-validation for the selection of the hyperparameters and an outer\n",
    "cross-validation for the evaluation of generalization performance of the\n",
    "refitted tuned model.\n",
    "\n",
    "In practice, we only need to embed the grid-search in the function\n",
    "`cross_validate` to perform such evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_results = cross_validate(\n",
    "    model_grid_search, data, target, cv=5, n_jobs=2, return_estimator=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_results = pd.DataFrame(cv_results)\n",
    "cv_test_scores = cv_results[\"test_score\"]\n",
    "print(\n",
    "    \"Generalization score with hyperparameters tuning:\\n\"\n",
    "    f\"{cv_test_scores.mean():.3f} \u00b1 {cv_test_scores.std():.3f}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This result is compatible with the test score measured with the string outer\n",
    "train-test split.\n",
    "\n",
    "However, in this case, we can apprehend the variability of our estimate of the\n",
    "generalization performance thanks to the measure of the standard-deviation of\n",
    "the scores measured in the outer cross-validation.\n",
    "\n",
    "Here is a schematic representation of the complete nested cross-validation\n",
    "procedure:\n",
    "\n",
    "![Nested cross-validation\n",
    "diagram](../figures/nested_cross_validation_diagram.png)\n",
    "\n",
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p>This figure illustrates the nested cross-validation strategy using\n",
    "<tt class=\"docutils literal\">cv_inner = KFold(n_splits=4)</tt> and <tt class=\"docutils literal\">cv_outer = KFold(n_splits=5)</tt>.</p>\n",
    "<p>For each inner cross-validation split (indexed on the right-hand side),\n",
    "the procedure trains a model on all the red samples and evaluate the quality\n",
    "of the hyperparameters on the green samples.</p>\n",
    "<p>For each outer cross-validation split (indexed on the left-hand side),\n",
    "the best hyper-parameters are selected based on the validation scores\n",
    "(computed on the green samples) and a model is refitted on the concatenation\n",
    "of the red and green samples for that outer CV iteration.</p>\n",
    "<p class=\"last\">The generalization performance of the 5 refitted models from the outer CV\n",
    "loop are then evaluated on the blue samples to get the final scores.</p>\n",
    "</div>\n",
    "\n",
    "In addition, passing the parameter `return_estimator=True`, we can check the\n",
    "value of the best hyperparameters obtained for each fold of the outer\n",
    "cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for cv_fold, estimator_in_fold in enumerate(cv_results[\"estimator\"]):\n",
    "    print(\n",
    "        f\"Best hyperparameters for fold #{cv_fold + 1}:\\n\"\n",
    "        f\"{estimator_in_fold.best_params_}\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is interesting to see whether the hyper-parameter tuning procedure always\n",
    "select similar values for the hyperparameters. If it is the case, then all is\n",
    "fine. It means that we can deploy a model fit with those hyperparameters and\n",
    "expect that it will have an actual predictive performance close to what we\n",
    "measured in the outer cross-validation.\n",
    "\n",
    "But it is also possible that some hyperparameters do not matter at all, and as\n",
    "a result in different tuning sessions give different results. In this case,\n",
    "any value will do. This can typically be confirmed by doing a parallel\n",
    "coordinate plot of the results of a large hyperparameter search as seen in the\n",
    "exercises.\n",
    "\n",
    "From a deployment point of view, one could also choose to deploy all the\n",
    "models found by the outer cross-validation loop and make them vote to get the\n",
    "final predictions. However this can cause operational problems because it uses\n",
    "more memory and makes computing prediction slower, resulting in a higher\n",
    "computational resource usage per prediction.\n",
    "\n",
    "In this notebook, we have seen how to evaluate the predictive performance of a\n",
    "model with tuned hyper-parameters using the nested cross-validation procedure."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}